Augmented reality display for material moving machines

ABSTRACT

A material moving machine including an implement, a camera, an augmented display, and a controller. The controller is programmed to store a three-dimensional model of underground features of terrain, capture an implement image comprising the implement and terrain, generate a superimposed image by superimposing corresponding portions of the implement image and the three-dimensional model of underground features, overlay a virtual trench on the superimposed image to generate an augmented reality overlay image, generate the augmented reality overlay image, and display the augmented reality overlay image on the augmented display.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/910,121, entitled “Augmented Reality Display for MaterialMoving Machines,” and filed on Mar. 2, 2018, which claims priority toU.S. Pat. App. No. 62/466,542, entitled “Augmented Reality Display forExcavators,” and filed on Mar. 3, 2017, the entireties of each of whichare incorporated by reference herein.

BACKGROUND

The present disclosure relates to material moving machines and, in someembodiments, to material moving machines including material movingimplements, such as excavators including excavating implements. Suchexcavators, for the purposes of defining and describing the scope of thepresent application, comprise an excavator boom and an excavator sticksubject to swing and curl, and an excavating implement that is subjectto swing and curl control with the aid of the excavator boom andexcavator stick, or other similar components for executing swing andcurl movement. For example, and not by way of limitation, many types ofexcavators comprise a hydraulically or pneumatically or electricallycontrolled excavating implement that can be manipulated by controllingthe swing and curl functions of an excavating linkage assembly of theexcavator. Excavator technology is, for example, well represented by thedisclosures of U.S. Pat. No. 8,689,471, which is assigned to CaterpillarTrimble Control Technologies LLC and discloses methodology forsensor-based automatic control of an excavator, US 2008/0047170, whichis assigned to Caterpillar Trimble Control Technologies LLC anddiscloses an excavator 3D laser system and radio positioning guidancesystem configured to guide a cutting edge of an excavator bucket withhigh vertical accuracy, and US 2008/0000111, which is assigned toCaterpillar Trimble Control Technologies LLC and discloses methodologyfor an excavator control system to determine an orientation of anexcavator sitting on a sloped site, for example.

BRIEF SUMMARY

According to the subject matter of the present disclosure, an excavatorcomprises a machine chassis, an excavating linkage assembly, anexcavating implement, a camera, a display, and guidance architecture,wherein the excavating linkage assembly comprises an excavator boom andan excavator stick that collectively define a plurality of linkageassembly positions. The excavating linkage assembly is configured toswing with, or relative to, the machine chassis. The excavator stick isconfigured to curl relative to the excavator boom. The excavatingimplement is mechanically coupled to the excavator stick. The camera ispositioned on the excavator such that a field of view of the cameraencompasses a view of the excavating implement and a terrain. Theguidance architecture comprises one or more dynamic sensors, one or morelinkage assembly actuators configured to actuate the excavating linkageassembly through the plurality of linkage assembly positions, and anarchitecture controller.

In accordance with embodiments of the present disclosure, thearchitecture controller is programmed to generate a camera image fromthe camera, the camera image comprising the excavating implement inrelation to the terrain, generate a virtual implement bar configured tobe disposed as on a leading edge of the excavating implement, generate avirtual target bar of a design surface configured to be disposed as partof the terrain at least partially based on a position of the excavatingimplement, overlay the virtual implement bar and the virtual target baron the camera image to generate an augmented reality overlay image, anddisplay the augmented reality overlay image on the display.

In accordance with other embodiments of the present disclosure, thearchitecture controller is programmed to generate a camera image fromthe camera, the camera image comprising the excavating implement inrelation to the terrain, generate a virtual trench configured to bedisposed as part of the terrain at least partially based on a positionof the excavating implement, superimpose the virtual trench on thecamera image to generate an augmented reality overlay image, and displaythe augmented reality overlay image on the display.

In accordance with yet other embodiments of the present disclosure, thearchitecture controller is programmed to store geometry data ofunderground features of the terrain, generate a camera image from thecamera, the camera image comprising the excavating implement in relationto the terrain, and generate a virtual trench configured to be disposedas part of the terrain at least partially based on the superimposedportion of a position of the excavating implement. The architecturecontroller is further programmed to generate a superimposed image of thevirtual trench on the camera image, overlay the underground featureswithin the virtual trench on the superimposed image to generate anaugmented reality overlay image, and display the augmented realityoverlay image on the display.

In accordance with embodiments of the present disclosure, a method ofoperating an excavator as shown and described herein is within the scopeof this disclosure.

According to the subject matter of the present disclosure, a materialmoving machine comprises a material moving implement, a camerapositioned on the material moving machine and comprising a field of viewthat encompasses the material moving implement and terrain in a workingarea of the material moving implement, an augmented display, and one ormore dynamic sensors configured to generate a position signalrepresenting a position of the material moving implement and anarchitecture controller. The architecture controller is programmed tostore a three-dimensional model of underground features of terrain in atleast the working area of the material moving implement, capture animplement image with the camera, the implement image comprising thematerial moving implement and terrain in at least the working area ofthe material moving implement, generate a superimposed image bysuperimposing corresponding portions of the implement image and thethree-dimensional model of underground features, overlay a virtualtrench on the superimposed image to generate an augmented realityoverlay image and display the augmented reality overlay image on theaugmented display.

In according with one other embodiment of the present disclosure, anexcavator comprises an excavating implement, a camera positioned on theexcavator and comprising a field of view that encompasses the excavatingimplement and terrain in the working area of the excavating implement,and control architecture comprising one or more dynamic sensorsconfigured to generate a position signal representing a position of theexcavating implement and an architecture controller. The architecturecontroller is programmed to store a three-dimensional model ofunderground features of terrain in at least the working area of thematerial moving implement, capture an implement image with the camera,the implement image comprising the excavating implement and terrain inat least the working area of the excavating implement, generate asuperimposed image by superimposing corresponding portions of theimplement image and the three-dimensional model of underground features,and overlay a virtual trench on the superimposed image to generate anaugmented reality overlay image, and display the augmented realityoverlay image on the augmented display.

In accordance with another embodiment of the present disclosure, amethod of operating a material moving machine utilizing an augmenteddisplay comprises disposing the material moving machine on terrain, thematerial moving machine comprising a material moving implement, acamera, the augmented display, and control architecture. The camera ispositioned on the excavator and comprises a field of view thatencompasses the material moving implement and terrain in a working areaof the material moving implement, and the control architecture comprisesone or more dynamic sensors configured to generate a position signalrepresenting a position of the material moving implement. The methodfurther comprises storing a three-dimensional model of undergroundfeatures of terrain in at least the working area of the material movingimplement, capturing an implement image with the camera, the implementimage comprising the material moving implement and terrain in at leastthe working area of the material moving implement, generating asuperimposed image by superimposing corresponding portions of theimplement image and the three-dimensional model of underground features,overlaying a virtual trench on the superimposed image to generate anaugmented reality overlay image and displaying the augmented realityoverlay image on the augmented display.

Although the concepts of the present disclosure are described hereinwith primary reference to the excavator illustrated in FIG. 1 as amaterial moving machine, it is contemplated that the concepts will enjoyapplicability to any type of material moving machine, regardless of itsparticular mechanical configuration. For example, and not by way oflimitation, the concepts may enjoy applicability to a backhoe loaderincluding a backhoe linkage.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The following detailed description of specific embodiments of thepresent disclosure can be best understood when read in conjunction withthe following drawings, where like structure is indicated with likereference numerals and in which:

FIG. 1 illustrates an excavator incorporating aspects of the presentdisclosure;

FIG. 2A is a schematic display view of an excavating implement of theexcavator of FIG. 1 in a position with respect to a virtual trench andincorporating aspects of the present disclosure;

FIG. 2B is a schematic display view of an excavating implement of theexcavator of FIG. 1 in a position with respect to a virtual implementbar and a virtual target bar of a design surface and incorporatingaspects of the present disclosure

FIG. 2C is a schematic display view of an excavating implement of theexcavator of FIG. 1 in a first position with respect to a virtual trenchand underground features and incorporating aspects of the presentdisclosure;

FIG. 3 is a schematic display view of an excavating implement of theexcavator of FIG. 1 in a second position with respect to the virtualtrench and underground features and incorporating aspects of the presentdisclosure;

FIG. 4 is a schematic display view of an excavating implement of theexcavator of FIG. 1 in a third position with respect to the virtualtrench and underground features and incorporating aspects of the presentdisclosure;

FIG. 5A is a flow chart illustrating a process that may be implementedby guidance architecture of the excavator of FIG. 1 to display anaugmented reality overlay image according to aspects of the presentdisclosure; and

FIG. 5B is a flow chart illustrating another process that may beimplemented by guidance architecture of the excavator of FIG. 1 todisplay an augmented reality overlay image according to aspects of thepresent disclosure.

DETAILED DESCRIPTION

The present disclosure relates to material moving machines configured toexecute material moving tasks such as those involving material movingoperations. For the purposes of the present disclosure, a materialmoving machine comprises a material moving implement and is designed toexcavate, distribute, smooth, or otherwise move a material using thematerial moving implement. Examples of such machines include, but arenot limited to, excavators, backhoe loaders, dozers, pavers, motorgraders, loaders, trenchers, scrapers, drills, crushers, draglines, orany type of machine that includes an implement for moving material.Contemplated materials include, but are not limited to, soil or othersurface-based earth materials, subterranean materials, includingmaterials to be mined, and construction aggregates, including, forexample, substrate materials and paving materials.

More particularly, the material moving machines may be excavatorsincluding components subject to adaptive control. For example, and notby way of limitation, many types of excavators typically have ahydraulically controlled material moving implement that can bemanipulated by a joystick or other means in an operator control stationof the machine, and is also subject to partially or fully automatedadaptive control. The user of the machine may control the lift, tilt,angle, and pitch of the implement. In addition, one or more of thesevariables may also be subject to partially or fully automated controlbased on information sensed or received by an adaptive environmentalsensor of the machine.

In the embodiments described herein, an excavator comprises a displayconfigured to display an augmented reality overlay image including avirtual implement bar, a virtual target bar of a design surface, and/ora virtual trench. In embodiments, the virtual trench comprisesunderground features. In other embodiments, the virtual trench may notcomprise underground features. The virtual trench is adjustablypositioned automatically based on a position of an excavating implement.For example, the position of the excavator implement is captured by acamera of the excavator and/or determined through signals from one ormore dynamic sensors, as described in greater detail further below.Alternatively or additionally, the virtual trench is adjustablypositioned automatically based on rotation of the excavator. Such anaugmented reality overlay image may be utilized by an excavator controland/or excavator operator to operate the excavator. For example, withrespect to the excavator operator's camera view of a real world, thephysical terrain image is augmented by virtual data to present theexcavator operator with an augmented reality camera view as theaugmented reality overlap image described in greater detail furtherbelow.

In embodiments, the virtual trench may be intended to representative ofan actual ditch to be dug. In other embodiments, the virtual trench maynot be intended to be representative of an actual ditch to be dug butrather be intended to visualize subsurface features including the designsurface in a manner that enables intuitive depth perception by utilizingan operator's understanding of a size and location of the excavatingimplement. Thus, the virtual trench may be representative of a trenchthat would be dug if the excavating implement were to dig verticallydown from a current location to the design surface and then pull intowards the excavator along the design surface. In some embodiments, thevirtual trench has a same width as a leading implement edge of theexcavating implement and is disposed with the leading implement edgeoriented in a direction towards the center of the excavator. Further, asthe excavating implement moves in to or away from the operator and/or asthe excavator rotates (i.e., slews), the virtual trench adjustablymoves, as described in greater detail further below.

In embodiments, generation of the virtual trench is at least partiallybased on terrain surface location to position and display linesrepresenting a top of the virtual trench on a terrain surface, whichterrain surface location may be estimated or measured. For example, aroll and pitch of the excavator may be projected to form a plane surfacein front of the excavator to utilize as the terrain surface location.Additionally or alternatively, a surface scanner may be used such as aLIDAR, RADAR, or photogrammetry based scanner for such terrain surfacelocation data. Further, GNSS-based vehicle mapping may used to measurethe terrain surface as the material moving machine moves around the site(also known as track mapping), the terrain surface may be estimated bymeasuring the movement of the implement, and/or at least one of terraindata from manual and aerial surveys may be used that is transferred toand stored by the guidance architecture of the excavator.

Referring initially to FIG. 1, an excavator 100 comprises a machinechassis 102, an excavating linkage assembly 104, an excavating implement114, a camera 124, a display 125, and guidance architecture 106. Inembodiments, the guidance architecture comprises control architectureresponsive to operator action through manual control of levers and/orlinkages and/or responsive to automatic action by a controller of theexcavator 100. The excavating linkage assembly 104 comprises anexcavator boom 108 and an excavator stick 110 that collectively define aplurality of linkage assembly positions. The camera 124 is positioned onthe excavator 100 such that a field of view of the camera 124encompasses a view of the excavating implement 114 and a terrain 126.

In embodiments, the excavator 100 comprises a boom dynamic sensor 120, astick dynamic sensor 122, an implement dynamic sensor 123, orcombinations thereof. The boom dynamic sensor 120 is positioned on theexcavator boom 108, the stick dynamic sensor 122 is positioned on theexcavator stick 110, and the implement dynamic sensor 123 is positionedon the excavating implement 114. The dynamic sensor 120, 122, 123 maycomprise a global navigation satellite system (GNSS) receiver, a globalpositioning system (GPS) receiver, a Universal Total Station (UTS) andmachine target, an inertial measurement unit (IMU), an inclinometer, anaccelerometer, a gyroscope, an angular rate sensor, a rotary positionsensor, a position sensing cylinder, or combinations thereof, or anysensor or combination of sensors that provide signals indicative of anoperational characteristic of a component of the material moving machinesuch as the excavator 100. For example, the dynamic sensor 120, 122, 123may comprise an IMU comprising a 3-axis accelerometer and a 3-axisgyroscope. The dynamic sensor 120, 122, 123 may include x-axis, y-axis,and z-axis acceleration values.

The excavating linkage assembly 104 may be configured to define alinkage assembly heading N and to swing with, or relative to, themachine chassis 102 about a swing axis S of the excavator 100. Theexcavator stick 110 is configured to curl relative to the excavator boom108. For example, the excavator stick 110 may be configured to curlrelative to the excavator boom 108 about a curl axis C of the excavator100. The excavator boom 108 and excavator stick 110 of the excavator 100illustrated in FIG. 1 are linked by a simple mechanical coupling thatpermits movement of the excavator stick 110 in one degree of rotationalfreedom relative to the excavator boom 108. In these types ofexcavators, the linkage assembly heading N will correspond to theheading of the excavator boom 108. However, the present disclosure alsocontemplates the use of excavators equipped with offset booms where theexcavator boom 108 and excavator stick 110 are linked by amultidirectional coupling that permits movement in more than onerotational degree of freedom. See, for example, the excavatorillustrated in U.S. Pat. No. 7,869,923 (“Slewing Controller, SlewingControl Method, and Construction Machine”). In the case of an excavatorwith an offset boom, the linkage assembly heading N will correspond tothe heading of the excavator stick 110.

The excavating implement 114 is mechanically coupled to the excavatorstick 110. For example, referring to FIG. 1, the excavating implement114 is mechanically coupled to the excavator stick 110 through animplement coupling 112. The excavating implement 114 may be mechanicallycoupled to the excavator stick 110 via the implement coupling 112 andconfigured to rotate about a rotary axis R. In an embodiment, the rotaryaxis R may be defined by the implement coupling 112 joining theexcavator stick 110 and the rotary excavating implement 114. In analternative embodiment, the rotary axis R may be defined by amultidirectional, stick coupling joining the excavator boom 108 and theexcavator stick 110 along the plane P such that the excavator stick 110is configured to rotate about the rotary axis R. Rotation of theexcavator stick 110 about the rotary axis R defined by the stickcoupling may result in a corresponding rotation of the rotary excavatingimplement 114, which is coupled to the excavator stick 110, about therotary axis R defined by the stick coupling.

The guidance architecture 106 comprises one or more dynamic sensors, oneor more linkage assembly actuators, and an architecture controllerprogrammed to execute the steps of a control scheme, such as a controlscheme 190 of FIG. 5A or a control scheme 200 of FIG. 5B, each of whichare described in greater detail further below. The guidance architecture106 may comprise a non-transitory computer-readable storage medium, suchas memory, comprising machine readable instructions. The guidancearchitecture 106 may also comprise a processor communicatively coupledto the non-transitory computer-readable storage medium and configured toexecute the machine readable instructions. The one or more linkageassembly actuators facilitate movement of the excavating linkageassembly 104. The one or more linkage assembly actuators comprise ahydraulic cylinder actuator, a pneumatic cylinder actuator, anelectrical actuator, a mechanical actuator, or combinations thereof.

In embodiments, and referring to FIG. 2A and a control scheme 190 ofFIG. 5A, the architecture controller is programmed to generate animplement image 130 from the camera 124 in step 192. As an example andnot as a limitation, the architecture controller is programmed tocapture an implement image as the capture image 130, the implement imagecomprising the excavating implement 114 and terrain in the working areaof the excavating implement 114. The implement image 130 comprises theexcavating implement 114 in relation to the terrain 126. Thearchitecture controller is further programmed to generate a virtualtrench 132 configured to be disposed as part of the terrain 126 at leastpartially based on a position of the excavating implement 114 in step194. In embodiments, the architecture controller may generate a signalindicative of the position and/or orientation of the excavatingimplement 114 and spacing with respect to terrain 126, as described ingreater detail further below, from the one or more dynamic sensors 120,122, 123. The architecture controller is further programmed to generatean image of the virtual trench 132 superimposed on the implement image130 to generate an augmented reality overlay image 134 in step 196, andto display the augmented reality overlay image 134 on the display 125 instep 198. As a non-limiting example, FIG. 2A illustrates a view of anaugmented reality overlay image 134 including the virtual trench 132without underground features. In other embodiments, as illustrated inFIG. 2C, the view of an augmented reality overlay image 134A may bedisplayed with underground features. The architecture controller may befurther programmed to generate the augmented reality overlay image toreveal underground features of the three-dimensional model within thevirtual trench 132 and to occlude underground features of thethree-dimensional model outside of the virtual trench 132.

For example, and referring to FIGS. 2C-4, the implement image 130comprises the excavating implement 114 in relation to the terrain 126and an augmented reality overlay image 134A, 134B, 134C displays avirtual trench 132 including underground features such as pipes 190, 192for display on the display 125. As a non-limiting example, FIG. 2Cillustrates a first view of such an augmented reality overlay image134A.

In embodiments, as illustrated in FIGS. 2A-2B, the augmented realityoverlay image 134, 134′ may be displayed without underground features.For example, in FIG. 2B, the augmented reality overlay image 134′ isdisplayed without the virtual trench 132 or the underground featureswhile displaying a virtual target implement bar 144 and a virtual targetbar 146 of a design surface 150, which are described in greater detailfurther below. In other embodiments, augmented reality overlay imagesmay be displayed with one or more portions of the virtual trench 132while omitting other one or more portions of the virtual trench 132, Forexample, a nearest trench boundary of the virtual trench 132 may beomitted from the display such that the virtual trench 132 appears toextend under the excavator 100 and outside of the view of the camera124. Sub-visualizations and/or expandable visualizations of suchaugmented reality overlay images may be displayed, such as a firstdisplay as illustrated in FIG. 2C of the virtual trench 132 includingunderground features that may be narrowed to a display of the virtualtrench 132 without underground features as displayed in FIG. 2A and/orthat may be narrowed to a display of the virtual target bar 146 withoutthe virtual trench 132 as displayed in FIG. 2B. Additionally, oralternatively, such displays may be expanded to add components andfeatures as described herein and/or may be displayed in a shared screenspace of the display 125.

As illustrated in FIG. 2C, the first view of an augmented realityoverlay image 134A may be displayed with underground features. Referringto FIGS. 2C-4 and the control scheme 200 of FIG. 5B, the architecturecontroller is programmed to store geometry data of underground featuresof the terrain 126 in step 202. In embodiments, alternatively oradditionally, the geometry data of underground features may be stored astwo-dimensional and/or three-dimensional geometry information. Further,additional underground feature information may be stored by thearchitecture controller that may or may not match a display of atwo-dimensional and/or three-dimensional model. As a non-limitingexample, underground feature information of an underground feature, suchas a pipe, may include two-dimensional and/or three-dimensionalcoordinates with respect to the start of the pipe and the end of thepipe. Further, the augmented reality overlay image 134A may include ashaded and/or curved extruded cylinder features to visually representthe pipe inserted between such coordinate points. In additionalembodiments, the geometry data of the underground features may includetwo-dimensional and/or three-dimensional coordinates of end points orcorners of the underground features, diameters of the undergroundfeatures such as pipes or cables, and/or other dimensions such asdimensions of sumps as underground features.

In step 204 of the control scheme 200 of FIG. 5B, the architecturecontroller is programmed to generate an implement image 130 from thecamera 124. As described above, the implement image 130 comprises theexcavating implement 114 in relation to the terrain 126. Further, instep 206, the architecture controller is programmed to generate avirtual trench 132 configured to be disposed as part of the terrain 126at least partially based on the position of the excavating implement114. For example, the architecture controller may generate a signalindicative of the position and/or orientation of the excavatingimplement 114 and spacing with respect to terrain 126, as described ingreater detail further below, from the one or more dynamic sensors 120,122, 123.

Further, boundaries of the virtual trench 132 including one or morewalls spaces defined between walls of the virtual trench 132 aredetermined such that, in step 208, the architecture controller isprogrammed to generate a superimposed image of the virtual trench 132 onthe implement image. In steps 210 and 212, respectively, thearchitecture controller is programmed to overlay at least a portion ofthe underground features within virtual trench 132 on the superimposedimage to generate an augmented reality overlay image 134A, 134B, 134C,and to display the augmented reality overlay image 134A, 134B, 134C onthe display 125.

In embodiments, the underground features displayed in the virtual trench132 may be truncated within a wall space defined by the virtual trench132. In other embodiments, portions of the underground features beyondthe boundaries of the virtual trench 132 may be displayed in a mannerthat differentiates such portions from the truncated underground featureportions displayed within the wall space as defined by the virtualtrench 132. As a non-limiting example, an underground feature mayinclude a pipe, and portions of the pipe that are displayed outside ofthe virtual trench 132 may include a different color and/or shading froma truncated portion of the pipe displayed within the virtual trench 132.Such visual differentiation may be indicative of a correspondingpositional difference with respect to placement of a portion of the pipeinside the virtual trench 132 and placement of one or more portions ofthe pipe disposed outside the virtual trench 132.

As will be described in greater detail further below, FIG. 2Cillustrates a first view of an augmented reality overlay image 134A.Further, FIG. 3 illustrates a second view of an augmented realityoverlay image 134B, and FIG. 4 illustrates a third view of an augmentedreality overlay image 134C. Referring to FIGS. 2C-4, in embodiments, atruncated portion 128 of the underground features is positioned withinthe virtual trench 132. The virtual trench 132 may be at least based inpart on a position and/or orientation of a leading implement edge 140 ofthe excavating implement 114 in relation to a longitudinally disposedsurface portion 142 of the terrain 126 that is disposed beneath theleading implement edge 140 of the excavating implement 114.

The augmented reality overlay image 134A, 134B, 134C may comprise avirtual implement bar 144 superimposed over the leading implement edge140 of the excavating implement 114. The augmented reality overlay image134A, 134B, 134C may further comprise a virtual target bar 146superimposed over a bottom-most horizontal portion of a front wall 154longitudinally extending from the longitudinally disposed surfaceportion 142 of the terrain 126 at a depth 163, 173, 183, as described ingreater detail further below.

The virtual target bar 146 of the design surface 150 may be configuredto adjust position in the augmented reality overlay image 134A, 134B,134C based on a position of the virtual implement bar 144 with respectto the terrain 126 and/or a predetermined depth of cut associated withthe design surface 150, for example. In embodiments, the design surface150 is indicative of a bottom of the virtual trench 132 that has afarthest-from-the-viewer boundary at which the virtual target bar 146 isdisposed. The design surface 150 may be a simple plane including adefined slope and/or direction, may include a road design, and/or mayinclude a predetermined depth of cut.

FIGS. 2C-4 respectively show different bar spacings 160, 170, 180between the virtual implement bar 144 and the virtual target bar 146below the terrain 126. For each of FIGS. 2-4, a respective bar spacing160, 170, 180 between the virtual implement bar 144 and the virtualtarget bar 146 is scalable with respect to a corresponding real worldspacing between the leading implement edge 140 of the excavatingimplement 114 and a pre-determined depth of cut below the longitudinallydisposed surface portion 142 of the terrain 126. Further, a respectiveadjustable length 162, 172, 182 of the pair of sidewalls 152A, 152B isdirectly proportional to the respective bar spacing 160, 170, 180, and arespective adjustable width 164, 174, 184 of the virtual target bar 146is inversely proportional to the respective bar spacing 160, 170, 180.Thus, a width 164, 174, 184 of the virtual trench 132 may be adjusted byan amount inversely proportional to the bar spacing, and a length 162,172, 182 of sidewalls 152A, 152B of the virtual trench 132 may beadjusted by an amount directly proportional to the bar spacing 160, 170,180.

Further, an adjustable depth 163, 173, 183 of the front wall 154 of thevirtual trench 132 is inversely proportional to the respectively barspacing 160, 170, 180. As an example, and not as a limitation, a depth163, 173, 183 of the virtual trench 132 may be adjusted by an amountinversely proportional to a bar spacing 160, 170, 180 between thevirtual target bar 146 and the virtual implement bar 144. Thus, thecloser the leading implement edge 140 is to the terrain 126 and virtualtarget bar 146, as indicated by bar spacings 160, 170, 180 between thevirtual implement bar 144 and the virtual target bar 146, the shorterthe virtual trench 132 appears in length with a greater depth and alonger width of the virtual target bar 146 at a rear portion of thevirtual trench 132 disposed furthest away from a viewer, for example.Alternatively, the farther the leading implement edge 140 is moved fromthe terrain 126 and virtual target bar 146, as indicated by bar spacings160, 170, 180 between the virtual implement bar 144 and the virtualtarget bar 146, the longer the virtual trench 132 appears in length witha shorter depth and a shorter width of the virtual target bar 146. Suchvisual adjustments may be indicative of changes in viewer perspectivedue to pixel width changes of the visual image, for example, and are notnecessarily indicative of real-world changes in dimensions associatedwith the components described herein or the virtual trench 132.

For example, with respect to FIG. 2C, the augmented reality overlayimage 134A includes a bar spacing 160, an adjustable length 162 of eachof the pair of sidewalls 152A, 152B, and an adjustable width 164 of thevirtual target bar 146. With respect to FIG. 3, in which the leadingimplement edge 140 is closer to the terrain 126 than in FIG. 2C, theaugmented reality overlay image 134B includes a bar spacing 170 that isless than the bar spacing 160 of FIG. 2C. Additionally, FIG. 3 includesan adjustable length 172 of each of the pair of sidewalls 152A, 152Bthat is less than the adjustable length 162 of FIG. 2, and an adjustablewidth 174 of the virtual target bar 146 that greater than the adjustablewidth 164 of FIG. 2C. Similarly, in FIG. 4, in which the leadingimplement edge 140 is closer to the terrain 126 than in FIGS. 2C-3, theaugmented reality overlay image 134C includes a bar spacing 180 that isless than the bar spacing 170 of FIG. 3. Further in FIG. 4, anadjustable length 182 of each of the pair of sidewalls 152A, 152B thatis less than the adjustable length 174 of FIG. 3, and an adjustablewidth 184 of the virtual target bar 146 that greater than the adjustablewidth 174 of FIG. 3. Thus, in each of augmented reality overlay images134A, 134B, 134C, as the virtual implement bar 144 moves closer to thevirtual target bar 146 on the terrain 126, the dynamically adjustableviews of the virtual trench 132 change to display a shortened adjustablelength 162, 172, 182 of the pair of sidewalls 152A, 152B, a greaterdepth adjustable depth 163, 173, 183 of the front wall 154, and agreater adjustable width 164, 174, 184 of the virtual target bar 146.The architecture controller and/or an excavator operator may set a rangeof dimensions for display with respect to the virtual trench 132 asdescribed herein. For the purposes of enhancing clarity in a 3Dvisualization of the virtual trench 132, for example, while the virtualtrench 132 may appear to have an increased depth visually as the virtualimplement bar 144 moves closer to the virtual target bar 146 on theterrain 126, a corresponding real-world depth illustrating undergroundfeatures (as described in greater detail further below) in the virtualtrench 132 may remain the same. As a non-limiting example, even though adepth 163 of the front wall 154 in FIG. 2C visually appears to be lessthan a depth 183 of the front wall 154 in FIG. 4, each of FIGS. 2-4 maycorrespond to a real-world depth of 5 feet from the terrain 126 to thevirtual target bar 146. With respect to such examples, undergroundfeatures within 5 feet from a surface of the terrain 126 are displayed,such as through illustrations as shown in FIGS. 2-4.

In embodiments, the virtual target bar 146 comprises a bottom-mostportion of the front wall 154 of the virtual trench 132. The virtualtrench 132 comprises a design surface 150 disposed beneath thelongitudinally disposed surface portion 142 at a depth 163, 173, 183, apair of sidewalls 152A, 152B with the design surface 150 disposedtherebetween, and the front wall 154 disposed between the pair ofsidewalls 152A, 152B and defining at the bottom-most portion the virtualtarget bar 146. The adjustable width 164, 174, 184 of the virtual targetbar 146 equals a width of the front wall 154 of the virtual trench 132.Boundaries of the virtual trench 132 including the design surface 150,the pair of sidewalls 152A, 152B, and the front wall 154 of the virtualtrench 132 may be depicted as dashed lines 148 in the augmented realityoverlay image 134A, 134B, 134C. In embodiments, the virtual target bar146 at a bottom boundary of the front wall 154 may be depicted as asolid and/or bolded line. The virtual trench 132, including the virtualtarget bar 146 and the boundaries of the virtual trench 132, isconfigured to be dynamically adjusted and dynamically displayed based onthe position and/or orientation of the virtual implement bar 144.

In an embodiment, with respect to a display of and within the augmentedreality overlay image 134A, 134B, 134C, the virtual implement bar 144comprises a solid red line. Further, the virtual target bar 146 maycomprise a solid green line. The virtual trench 132 may be defined by aplurality of dashed lines 148 that are red in color or other suitablehighlighting and differentiating color. It should be understood thatother colors for such displayable components are within the scope ofthis disclosure.

In embodiments, and within the virtual trench 132, underground featuresare displayed that adjust along with the dynamic adjustment of thevirtual trench 132 in the augmented reality overlay images 134A, 134B,134C and that are truncated at walls of the virtual trench 132 to enableenhanced depth perception. The underground features may comprise, forexample, portions of pipes and/or cables or like underground componentstruncated to be defined and positioned with the boundaries of thevirtual trench 132 at least partially based on the stored geometry dataof underground features. To further enhance depth perception, display ofthe underground features may be indicative of a spacing and/or distanceof the underground features with respect to the wall boundaries of thevirtual trench 132. For example, in FIG. 2C, a pipe 190 is displayed asnearer a surface of the terrain 126 than a pipe 192, while the pipe 192is displayed as distanced farther from the surface of the terrain 126and closer to the design surface 150. In embodiments, the virtual trench132 may be displayed as described herein without a display ofunderground features.

The underground features may comprise at least one of a pipe 190 andcable disposed below the terrain. The virtual trench 132 may comprise atleast a portion of the at least one of a pipe 190 and cable in theaugmented reality overlay image 134A, 134B, 134C. In embodiments, theunderground features comprise at least a pair of pipes 190, 192 disposedbelow the terrain 126. A first pipe 190 of the pair of pipes 190, 192may be disposed above and at angle to a second pipe 192 of the pair ofpipes 190, 192 as shown in the augmented reality overlay image 132A ofFIG. 2C. For example, the first pipe 190 of the pair of pipes 190, 192may be disposed perpendicular to the second pipe 192 of the pair ofpipes 190, 192. One pipe of the pair of pipes 190, 192, such as the pipe190 in FIG. 2C, may be disposed at an intersecting angle with respect tothe front wall 154 of the virtual trench 132. The display of this anglemay change depending on the display of the dimensions of the virtualtrench 132 in the augmented reality overlay image 134A, 134B, 134C.Further, the virtual trench 132 may comprise at least a portion of eachof the pair of pipes 190, 192 in the augmented reality overlay image134A, as shown in FIG. 2C, in which each of the pair of pipes 190, 192is displayed with a different color with respect to one another. Forexample, one pipe of the pair of pipes 190, 192, such as the pipe 190,may comprise a yellow color, and the other pipe of the pair of pipes190, 192, such as the pipe 192, may comprise a green color.

It is contemplated that the embodiments of the present disclosure mayassist to provide a more efficient method of excavation planning on sitein a manner that minimizes a risk of human error in planning anexcavation. For example, the embodiments described herein may assist tominimize a risk of human error in estimating and/or being aware of apositioning of underground features when excavating in real-time. Thecontroller of the excavator or other control technologies may beimproved with such augmented reality overlay images as described hereinsuch that processing systems in connection with the augmented realityoverlay images are enhanced and improved with respect to efficiency,speed, safety, production, and accuracy.

A signal may be “generated” by direct or indirect calculation ormeasurement, with or without the aid of a sensor.

For the purposes of describing and defining the present invention, it isnoted that reference herein to a variable being “based on” a parameteror another variable is not intended to denote that the variable isexclusively based on the listed parameter or variable. Rather, referenceherein to a variable that is “based on” a listed parameter is intendedto be open ended such that the variable may be based on a singleparameter or a plurality of parameters.

It is also noted that recitations herein of “at least one” component,element, etc., should not be used to create an inference that thealternative use of the articles “a” or “an” should be limited to asingle component, element, etc.

It is noted that recitations herein of a component of the presentdisclosure being “configured” or “programmed” in a particular way, toembody a particular property, or to function in a particular manner, arestructural recitations, as opposed to recitations of intended use. Morespecifically, the references herein to the manner in which a componentis “configured” or “programmed” denotes an existing physical conditionof the component and, as such, is to be taken as a definite recitationof the structural characteristics of the component.

Having described the subject matter of the present disclosure in detailand by reference to specific embodiments thereof, it is noted that thevarious details disclosed herein should not be taken to imply that thesedetails relate to elements that are essential components of the variousembodiments described herein, even in cases where a particular elementis illustrated in each of the drawings that accompany the presentdescription. Further, it will be apparent that modifications andvariations are possible without departing from the scope of the presentdisclosure, including, but not limited to, embodiments defined in theappended claims. More specifically, although some aspects of the presentdisclosure are identified herein as preferred or particularlyadvantageous, it is contemplated that the present disclosure is notnecessarily limited to these aspects.

It is noted that one or more of the following claims utilize the term“wherein” as a transitional phrase. For the purposes of defining thepresent disclosure, it is noted that this term is introduced in theclaims as an open-ended transitional phrase that is used to introduce arecitation of a series of characteristics of the structure and should beinterpreted in like manner as the more commonly used open-ended preambleterm “comprising.” Likewise, one or more of the following claims utilizethe term “based on,” which similarly is an open-ended phrase that shouldbe interpreted in like manner as the more commonly used open-endedpreamble term “comprising.”

What is claimed is:
 1. A material moving machine comprising: a materialmoving implement; a camera positioned on the material moving machine andcomprising a field of view that encompasses the material movingimplement and terrain in a working area of the material movingimplement; an augmented display; one or more dynamic sensors configuredto generate a position signal representing a position of the materialmoving implement; and an architecture controller programmed to store athree-dimensional model of underground features of terrain in at leastthe working area of the material moving implement, capture an implementimage with the camera, the implement image comprising the materialmoving implement and terrain in at least the working area of thematerial moving implement, generate a superimposed image bysuperimposing corresponding portions of the implement image and thethree-dimensional model of underground features, overlay a virtualtrench on the superimposed image to generate an augmented realityoverlay image, and display the augmented reality overlay image on theaugmented display.
 2. The material moving machine of claim 1, whereinthe architecture controller is programmed to generate the augmentedreality overlay image to reveal underground features of thethree-dimensional model within the virtual trench and to occludeunderground features of the three-dimensional model outside of thevirtual trench.
 3. The material moving machine of claim 1, wherein thearchitecture controller is programmed to generate the virtual trenchbased on a position of a leading implement edge of the material movingimplement in relation to a longitudinally disposed surface portion ofthe terrain disposed beneath the leading implement edge of the materialmoving implement.
 4. The material moving machine of claim 3, wherein thearchitecture controller is programmed to generate the augmented realityoverlay image to comprise: a virtual implement bar superimposed over theleading implement edge of the material moving implement; and a virtualtarget bar superimposed over the longitudinally disposed surface portionof the terrain and configured to adjust position in the augmentedreality overlay image based on a position of the virtual implement barwith respect to the virtual target bar on the terrain.
 5. The materialmoving machine of claim 4, wherein: the virtual implement bar comprisesa solid red line; the virtual target bar comprises a bottom-most portionof the virtual trench as a solid green line; and the virtual trench isdefined by a plurality of dashed red lines.
 6. The material movingmachine of claim 4, wherein the architecture controller is programmed togenerate the virtual trench such that the virtual target bar comprises abottom-most portion of the virtual trench and the virtual trenchcomprises: an underground surface; a pair of sidewalls disposed betweenthe underground surface; and a front wall disposed between the pair ofsidewalls and below the virtual target bar.
 7. The material movingmachine of claim 6, wherein the architecture controller is programmed togenerate: a bar spacing between the virtual implement bar and thevirtual target bar that is scalable with respect to a corresponding realworld spacing with the leading implement edge of the material movingimplement and the longitudinally disposed surface portion of theterrain; an adjustable width of the virtual target bar that is inverselyproportional to the bar spacing; and an adjustable length of the pair ofsidewalls that is directly proportional to the bar spacing.
 8. Thematerial moving machine of claim 7, wherein the adjustable width of thevirtual target bar equals a width of the front wall of the virtualtrench.
 9. The material moving machine of claim 6, wherein thearchitecture controller is programmed to depict boundaries of theunderground surface, the pair of sidewalls, and the front wall of thevirtual trench as dashed lines in the augmented reality overlay image.10. The material moving machine of claim 3, wherein the architecturecontroller is programmed to adjust the virtual trench based on theposition of the virtual implement bar.
 11. The material moving machineof claim 1, wherein the architecture controller is programmed to disposeat least a pair of pipes comprising the underground features below theterrain.
 12. The material moving machine of claim 11, wherein thearchitecture controller is programmed to dispose a first pipe of thepair of pipes above and at angle to a second pipe of the pair of pipes.13. The material moving machine of claim 12, wherein the architecturecontroller is programmed to dispose the first pipe of the pair of pipesperpendicular to the second pipe of the pair of pipes.
 14. The materialmoving machine of claim 13, wherein the architecture controller isprogrammed to: generate the virtual trench to comprise at least aportion of each of the pair of pipes in the augmented reality overlayimage; and display each of the pair of pipes with a different color withrespect to one another.
 15. The material moving machine of claim 13,wherein the architecture controller is programmed to: generate thevirtual trench to comprise a front wall; and dispose one pipe of thepair of pipes an intersecting angle with respect to the front wall ofthe virtual trench.
 16. An excavator comprising: an excavatingimplement; a camera positioned on the excavator and comprising a fieldof view that encompasses the excavating implement and terrain in theworking area of the excavating implement; and control architecturecomprising one or more dynamic sensors configured to generate a positionsignal representing a position of the excavating implement and anarchitecture controller programmed to store a three-dimensional model ofunderground features of terrain in at least the working area of thematerial moving implement, capture an implement image with the camera,the implement image comprising the excavating implement and terrain inat least the working area of the excavating implement, generate asuperimposed image by superimposing corresponding portions of theimplement image and the three-dimensional model of underground features,overlay a virtual trench on the superimposed image to generate anaugmented reality overlay image, and display the augmented realityoverlay image on the augmented display.
 17. A method of operating amaterial moving machine utilizing an augmented display comprising:disposing the material moving machine on terrain, the material movingmachine comprising a material moving implement, a camera, the augmenteddisplay, and control architecture, wherein (i) the camera is positionedon the excavator and comprises a field of view that encompasses thematerial moving implement and terrain in a working area of the materialmoving implement, and (ii) the control architecture comprises one ormore dynamic sensors configured to generate a position signalrepresenting a position of the material moving implement; storing athree-dimensional model of underground features of terrain in at leastthe working area of the material moving implement; capturing animplement image with the camera, the implement image comprising thematerial moving implement and terrain in at least the working area ofthe material moving implement; generating a superimposed image bysuperimposing corresponding portions of the implement image and thethree-dimensional model of underground features; overlaying a virtualtrench on the superimposed image to generate an augmented realityoverlay image; and displaying the augmented reality overlay image on theaugmented display.
 18. The method of claim 17, further comprising:operating the material moving machine utilizing the control architectureto move the material moving implement with respect to the terrain basedon the augmented reality overlay image.
 19. The method of claim 18,further comprising: generating the virtual trench based on a position ofa leading implement edge of the material moving implement in relation toa longitudinally disposed surface portion of the terrain disposedbeneath the leading implement edge; and generating the augmented realityoverlay image comprising a virtual implement bar superimposed of aleading implement edge of the material moving implement, and virtualtarget bar superimposed over a longitudinally disposed surface portionof the terrain.
 20. The method of claim 19, further comprising:adjusting a position of the virtual target bar in the augmented realityoverlay image based on a position of the virtual implement bar withrespect to the virtual target bar on the terrain, adjusting a depth ofthe virtual trench by an amount inversely proportional to a bar spacingbetween the virtual target bar and the virtual implement bar; adjustinga width of the virtual trench by an amount inversely proportional to thebar spacing; and adjusting a length of sidewalls of the virtual trenchby an amount directly proportional to the bar spacing.